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ABSTRACT 



A computer-driven system aids operator positioning of a 
cursor by integrating eye gaze and manual operator input, 
thus reducing pointing time and operator fatigue. A gaze 
tracking apparatus monitors operator eye orientation while 
the operator views a video screen. Concurrently, the com- 
puter monitors an input device, such as a mouse, for 
mechanical activation by the operator. According to the 
operator's eye orientation, the computer calculates the 
operator's gaze position. Also computed is a gaze area, 
comprising a sub-region of the video screen that includes the 
gaze position. This region, for example, may be a circle of 
sufficient radius to include the point of actual gaze with a 
certain likelihood. When the computer detects mechanical 
activation of the operator input device, it determines an 
initial cursor display position within the current gaze area. 
This position may be a predetermined location with respect 
to the gaze area, such as a point on the bottom of the gaze 
area periphery. A different approach uses the initial mechani- 
cal activation of the input device to determine the direction 
of motion, and sets the initial display position on the 
opposite side of the gaze area from this motion so that 
continued movement of the input device brings the cursor to 
the gaze position in a seamless transition between gaze and 
manual input. After displaying the cursor on the video screen 
at the initial display position, the cursor is thereafter posi- 
tioned manually according to the operator's use of the input 
device, without regard to gaze. .. 

58 Claims, 3 Drawing Sheets 
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INTEGRATED GAZE/MANUAL CURSOR 
POSITIONING SYSTEM 

BACKGROUND OF THE INVENTION 

1 . Field of the Invention 

The present invention relates to video cursor positioning 
systems. More particularly, the invention concerns a system 
for positioning a cursor on a video screen, in which pointing 
time and operator fatigue are reduced by integrating eye 
gaze tracking and manual operator input. 

2. Description of the Related Art 

In human-computer interaction, one of the most basic 
elements involves selecting a target using a pointing device. 
Target selection is involved in opening a file with a mouse 
"click", activating a world wide web link, selecting a menu 
item, redefining a typing or drawing insertion position, and 
other such operations. Engineers and scientists have devel- 
oped many different approaches to target selection. One of 
the most popular target selection devices is the computer 
mouse. Although computer mouses are practically essential 
with today's computers, intense use can cause fatigue and 
repetitive motion injury. 

Despite these limitations, further improvement of mouse- 
activated target selection systems has been difficult. One 
interesting idea for possible improvement uses eye gaze 
tracking instead of mouse input. There are several known 
techniques for monitoring eye gaze. One approach senses 
the electrical impulses of eye muscles to determine eye gaze. 
Another approach magnetically senses the position of spe- 
cial user- worn contact lenses having tiny magnetic coils. 
Still another technique, called "corneal reflection", calcu- 
lates eye gaze by projecting an invisible beam of light 
toward the eye, and monitoring the angular difference 
between pupil position and reflection of the light beam. 

With these types of gaze tracking systems, the cursor is 
positioned on a video screen according to the calculated 
gaze of the computer operator. A number of different tech- 
niques have been developed to select a target in these 
systems. In one example, the system selects a target when it 
detects the operator fixating at the target for a certain time. 
Another way to select a target is when the operator's eye 
blinks. One problem with these systems is that humans use 
their the eyes naturally as perceptive, not manipulative, body 
parts. Eye movement is often outside conscious thought, and 
it can be stressful to carefully guide eye movement as 
required to accurately use these target selection systems. For 
many operators, controlling blinking or staring can be 
difficult, and may lead to inadvertent and erroneous target 
selection. Thus, although eye gaze is theoretically faster than 
any other body part, the need to use unnatural selection (e.g., 
by blinking or staring) limits the speed advantage of gaze 
controlled pointing over manual pointing. 

Another limitation of the foregoing systems is the diffi- 
culty in making accurate and reliable eye tracking systems. 
Only relatively large targets can be selected by gaze con- 
trolling pointing techniques, because of eye jitter and other 
inherent difficulties in precisely monitoring eye gaze. 

Consequently, known gaze-based target selection systems 
are not adequate for some applications due to certain 
unsolved problems. 

SUMMARY OF THE INVENTION 

Broadly, the present invention concerns a system for 
positioning a cursor on a video screen, where eye gaze 
information and manual operator input are integrated to 
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reduce pointing time and associated operator fatigue. With 
the invention, pointing and target selection are perceived as 
a manual task, but they are aided with gaze tracking. Gaze 
aided tracking is used to dynamically redefine a home 

5 position of the pointing cursor in the vicinity of the target of 
interest, to reduce the subsequent movement distance 
needed to manually select the target. 

A gaze tracking apparatus monitors eye orientation of a 
computer operator as the operator views a video screen. 

30 Concurrently, the computer monitors an operator input 
device for activation by the operator. The manual operator 
input device comprises a mechanically activated device, 
such as a mouse, keyboard, isometric pointing transducer, 
trackball, and the like. According to the operator's eye 

15 orientation, the computer measures the point of the opera- 
tor's gaze, called "gaze position." The computer also cal- 
culates a region of video screen surrounding the operator's 
gaze position. This region, for example, may be a circle that 
is calculated to include the operator's "actual gaze point" (as 

20 opposed to measured gaze position) with a certain probabil- 
ity. 

When the computer detects mechanical activation of the 
operator input device, it determines an "initial display posi- 
tion" within the gaze area, at which to display the cursor. As 

25 one example, the initial display position may be a predeter- 
mined location with respect to the gaze area, such as a point 
on the bottom or top of the gaze area periphery. A different 
approach is to determine an initial direction of motion from 
the activation of the input device, and then set the initial 

30 display position on the opposite side of the gaze area from 
this motion. Therefore, continued movement of the input 
device in the same direction will direct the cursor to the 
measured gaze position, providing a seamless transition 
from gaze to manual input. 

After displaying the cursor on the video screen at the 
initial display position, the cursor is thereafter positioned 
manually according to the operator's use of the input device. 
If the input device becomes inactive for a prescribed time, 

40 and is thereafter reactivated, the computer determines the 
difference between the cursor's position when inactivity 
began and the operator's present gaze position. If this 
difference is small, the cursor continues to appear where it 
was left when the input device became inactive. However, if 

45 there is a large difference, a new initial display position is 
calculated, considering gaze area and input device move- 
ment as mentioned above, and the cursor is repositioned 
there. 

Accordingly, in one embodiment, the invention may be 

50 implemented to provide a computer-driven method to posi- 
tion a cursor on a video screen, where eye gaze information 
and manual operator input are integrated to reduce pointing 
time and associated operator fatigue. In another 
embodiment, the invention may be implemented to provide 

55 an apparatus, such as a digital processing apparatus, con- 
figured to position a cursor on a video screen, where eye 
gaze information and manual operator input are integrated to 
reduce pointing time and associated operator fatigue. In still 
another embodiment, the invention may be implemented to 

60 provide a signal-bearing medium tangibly embodying a 
program of machine-readable instructions executable by a 
digital data processing apparatus to perform method steps 
for positioning a cursor on a video screen, where eye gaze 
information and manual operator input are integrated to 

65 reduce pointing time and associated operator fatigue. 

The invention affords its users with a number of distinct 
advantages. Generally, the invention enables computer users 
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to select video screen targets more quickly than with a 
manual pointing device alone. Since the user's target of 
interest is always inside the gaze area, and the cursor is 
initially presented near the gaze area, the user does not have 
to move the cursor over large distances of the computer 
screen. Advantageously, the pointing cursor always appears 
near where it needs to be. The faster speed and use of gaze 
helps reduces an operator's reliance on a computer mouse 
and keyboard, helping to avoid fatigue and possible repeti- 
tive motion injuries. In addition, the integration of gaze and 
manual input is seamless and therefore especially conve- 
nient to the operator. The invention also provides a number 
of other advantages and benefits, which should be apparent 
from the following description of the invention. 

BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 is a block diagram of the hardware components 
and interconnections of an integrated gaze/manual cursor 
management system in accordance with the invention. 

FIG. 2 is a block diagram of a digital data processing 
machine in accordance with the invention. 

FIG. 3 shows an exemplary signal-bearing medium in 
accordance with the invention. 

FIG. 4 is a flowchart showing computer-implemented 
process steps used to implement one particular embodiment 
of integrated gaze/manual cursor management system 
according to the invention. 

DETAILED DESCRIPTION 

The nature, objects, and advantages of the invention will 
become more apparent to those skilled in the art after 
considering the following detailed description in connection 
with the accompanying drawings. As mentioned above, the 
invention concerns a cursor positioning system that 
increases pointing speed and reduces operator fatigue by 
integrating eye gaze tracking and manual operator input. 

HARDWARE COMPONENTS & 
INTERCONNECTIONS 

System Overview 

One aspect of the invention concerns an integrated gaze/ 
manual cursor positioning system, which may be embodied 
by various hardware components and interconnections as 
illustrated by the system 100 of FIG. 1. Generally, the 
system 100 includes a computer 102, a gaze tracking appa- 
ratus 104, a user input device 106, and a display 120. The 
system 100 is used by a "user", also called an "operator" (not 
shown). 

Gaze Tracking Apparatus 

The gaze tracking apparatus 104 is a device for monitor- 
ing the eye gaze of the computer operator. The device 104 
may use many different techniques to monitor eye gaze, 
depending upon the particular needs of the application. As 
one example, the device 104 may employ one or more of the 
following: 

1. Electro-Oculography. This technique places skin elec- 
trodes around the eye, and records potential differences, 
representative of eye position. 

2. Corneal Reflection. This technique directs an infrared 
light beam at the operator's eye and measures the 
angular difference between the operator's mobile pupil 
and the stationary light beam reflection. 

3. Lumbus, Pupil, and Eyelid Tracking. These techniques 
include scanning the eye region with an apparatus such 
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as a television camera or other scanner, and analyzing 
the resultant image. 
4. Contact Lens. These techniques use. some device 
attached to the eye with a specially manufactured 

S contact lens. With the "optical lever", for example, one 
or more plane minor surfaces ground on the lens reflect 
light from a light source to a photographic plate or 
photocell or quadrant detector array. Another approach 
uses a magnetic sensor in conjunction with contact 

10 lenses with implanted magnetic coils. 

A number of different gaze tracking approaches are surveyed 
in the following reference, which is incorporated herein in 
its entirety: Young et al., "Methods & Designs: Survey of 
Eye Movement Recording Methods", Behavior Research 

15 Methods & Instrumentation, 1975, Vol. 7(5), pp. 397-429. 
Ordinarily skilled artisans, having the benefit of this 
disclosure, will also recognize a number of different devices 
suitable for use as the apparatus 104. 

As a specific example of one gaze tracking approach for 

20 use in the present invention, reference is made to the 
following patents, incorporated herein in their entirety: (1) 
U.S. Pat. No. 4,836,670 to Hutchison, issued Jun. 6, 1989 
and entitled "Eye Movement Detector", (2) U.S. Pat. No. 
4,950,069 to Hutchison, issued Aug. 21, 1990 and entitled 

25 "Eye Movement Detector With Improved Calibration and 
Speed", and (3) U.S. Pat. No. 4,595,990 to Garwin et al., 
issued Jun. 17, 1986 and entitled "Eye Controlled Informa- 
tion Transfer". Also, although the invention's gaze tracking 
apparatus 104 may be a custom product, a commercially 

30 available product may be used instead, such as the EyeTrac 
Series 4000 product by Applied Science Labs, or the Eye- 
Gaze system by LC Technology. 

Although the software programming associated with the 
gaze tracking apparatus 104 may be included with the 

35 apparatus 104 itself, the particular example of FIG. 1 shows 
the associated software implemented in the gaze tracking 
module 118, described below. The module 118 may be 
included solely in the computer 102, in the apparatus 104, or 
in a combination of the two, depending upon the particular 

40 application. 

Advantageously, the invention is capable of accurate 
operation with inexpensive, relatively low-resolution gaze 
tracking apparatuses. For instance, significant benefits can 
be gained with gaze tracking accuracy ±3 degrees, which is 

45 a low error requirement for gaze tracking systems. With this 
level of permissible error, the gaze tracking apparatus 104 
may comprise an inexpensive video camera, many of which 
are known and becoming increasingly popular for use in 
computer systems. 

50 User Input Device(s) 

The user input device 106 comprises an operator input 
device with an element sensitive to pressure, physical 
contact, or other manual activation by a human operator. 
This is referred to as "manual" input that "mechanically" 

55 activates the input device 106, in contrast to gaze input from 
the gaze tracking apparatus. As an example, the device 106 
may include one or more of the following: a computer 
keyboard, a mouse, "track-ball", a foot-activated switch or 
trigger, pressure-sensitive transducer stick such as the IBM 

60 TRACKPOINT product, tongue activated pointer, and/or 
another mechanically activated device. In the particular 
embodiment illustrated in FIG. 1, a keyboard 108 and mouse 
110 are shown. Although the software programming asso- 
ciated with the input device 106 may be included with the 

65 device 106, the particular example of FIG. 1 shows the 
necessary input device software implemented in the user 
input module 116, described below. The module 116 may be 
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included solely in the computer 102, the input device 106, or the user input module 116 and input device 106) with eye 

a combination of the two, depending upon the particular gaze input (from the gaze tracking apparatus 104 and gaze 

application. tracking module 118). The integration module 114 applies 

Display certain criteria to input from the components 104/106 to 

The display 120 provides an electronic medium for opti- s determine how the cursor is shown on the display 120. 
cally presenting text and graphics to the operator. The 

display 120 may be implemented by any suitable computer Exemplary Digital Data Processing Apparatus 

display with sufficient ability to depict graphical images Another aspect of the invention concerns a digital data 

including a cursor. For instance, the display 120 may employ processing apparatus, which may be used to implement one 

a cathode ray tube, liquid crystal diode screen, light emitting 10 or more components of the system 100. This apparatus may 

diode screen, or another suitable video apparatus. The De embodied by various hardware components and 

images of the display 120 are determined by signals from the interconnections, as exemplified in FIG. 2 by the digital data 

video module 122, described below. The display 120 may processing apparatus 200. The apparatus 200 includes a 

also be referred to by other names, such as video display, processor 202, such as a microprocessor or other processing 

video screen, display screen, video monitor, display monitor, 15 mac hine, coupled to a storage 204. In the present example, 

etc. The displayed cursor may comprise an arrow, bracket, ^ stor age 204 includes a fast-access storage 206, as well as 

short line, dot, cross-hair, or any other image suitable for nonvolatile storage 208. The fast-access storage 206 may 

selecting targets, positioning an insertion point for text or comprise random access memory, and may be used to store 

graphics, etc. me programming instructions executed by the processor 

Computer 20 2 02. The nonvolatile storage 208 may comprise, for 

The computer 102 includes one or more application example, one or more magnetic data storage disks such as a 

programs 112, a user input module 116, a gaze tracking " hard drive ^ electronic read-only memory, magnetic tape, 

module 118, an integration module 114, and a video module CD-ROM, or any other suitable storage device. The appa- 

122. The computer 102 may be a new machine, or one ratus 2 00 also includes an input/output 210, such as a line, 

selected from any number of different products such as a 25 bus, cable, electromagnetic link, or other means for 

known personal computer, computer workstation, main- exchanging data with the processor 202. 

frame computer, or another suitable digital data processing D ^ ^ jfic for oi description7 ord inarily 

™™ CXamP ' t\ C ° m r u ^ ^ =™ artisans (having the benefit of this disclosure) will 

THINKPAD computer Although such a computer clearly ^ ^ ^ g discus&ed aboye b£ . 

includes a number of other components addition ^ose of 30 mented m a machine of constructioilj without 

FIG 1, these components are omitted from FIG. 1 for ease from ^ scQpe Qf ^ invention> M a specific 

ol lustration, example, one of the components 206, 208 may be elimi- 

The video module 122 comprises a product that generates n th e storage 204 may be provided 

video signals representing .images. TTiese signals are com- OQ . board ^ essor 2Q2 of eyen ided extcrnaJ1 tQ 

patible with the display 120, and cause the display 120 to 35 ^ aratus 200 
show the corresponding images. The video module 122 may 

be provided by hardware, software, or a combination. As a OPERATION 
more specific example, the video module 122 may be a 

video display card, such as an SVGA card. In addition to the hardware environment described above, 
Tne application programs 112 comprise various programs *o a different aspect of the invention concerns a computer- 
running on the computer 102, and requiring operator input implemented method for cursor positioning by integrating 
from time to time. This input may include text (entered via S aze tracking a nd manual operator input. As an example, 
the keyboard 108) as well as positional and target selection this mcthod ma y bc implemented in the particular hardware 
information (entered using the mouse 110). The positional environment discussed above, 
information positions a cursor relative to images supplied by 45 Sienal-Bearine Media 
the application program. The target selection information 

selects a portion of the displayed screen image identified by In the context of FIGS. 1-2, such a method may be 

the cursor position at the moment the operator performs an implemented, for example, by operating the computer 102 

operation such as a mouse "click". Examples of application (FIG. 1), as embodied by a digital data processing apparatus 

programs 112 include commercially available programs 50 200 (FIG. 2), to execute a sequence of machine-readable 

such as database programs, word processing, financial instructions. These instructions may reside in various types 

software, computer games, computer aided design, etc. of signal-bearing media. In this respect, one aspect of the 

The user input module 116 comprises a software module present invention concerns a programmed product, compris- 

configured to receive and interpret signals from the input ing signal-bearing media tangibly embodying a program of 

device 106. As a specific example, the module 116 may 55 machine -readable instructions executable by a digital data 

include a mouse driver that receives electrical signals from processor to perform a method to perform integrated gaze/ 

the mouse 110 and provides an x-y output representing manual cursor positioning. 

where the mouse is positioned. Similarly, the gaze tracking This signal-bearing media may comprise, for example, 

module 118 comprises a software module configured to RAM (not shown) contained within the computer 102, as 

receive and interpret signals from the gaze tracking appa- 60 represented by the fast- access storage 206 for example, 

ratus 104. As a specific example, the module 118 may Alternatively, the instructions may be contained in another 

include a program that receives electrical signals from the signal-bearing media, such as a magnetic data storage dis- 

apparatus 104 and provides an x-y output representing a kette 300 (FIG. 3), directly or indirectly accessible by the 

point where the operator is calculated to be gazing, called computer 102. Whether contained in the diskette 300, the 

the "gaze position". 65 computer 102, or elsewhere, the instructions may be stored 

As explained in greater detail below, the integration on a variety of machine-readable data storage media, such as 

module 114 serves to integrate manual operator input (from DASD storage (e.g., a conventional "hard drive" or a RAID 
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array), magnetic tape, electronic read-only memory (e.g., received and includes the operator's actual gaze point. As 

ROM, EPROM, or EEPROM), an optical storage device one example, the gaze area may be calculated to include the 

(e.g. CD-ROM, WORM, DVD, digital optical tape), paper actual gaze point with a prescribed degree of probability, 

"punch" cards, or other suitable signal-bearing media such as 95%. In other words, the gaze area in this example 

including transmission media such as digital and analog and s comprises a region in which the user's actual gaze point is 

communication links and wireless. In an illustrative embodi- statistically likely to reside, considering the measured gaze 

ment of the invention, the machine-readable instructions position and predicted or known system noise. Thus, the 

may comprise software object code, compiled from a lan- gaze area's shape and size may change according to cursor 

guage such as "C", etc. position on the display 120, because some areas of the 

10 display 120 may be associated with greater noise than 

Overall Sequence of Operation others. As a further example, the gaze area may comprise a 

Introduction c * rc * e °^ su ® c ^ ent raa "i us to include the actual gaze point 

* . - 4 . , AtxtXi M1 A 4 within a prescribed probability, such as three standard devia- 

FIG. 4 shows a sequence of method steps 400 to illustrate ^ f <sigma » )t In mis embodiment, the circle representing 

one example of me method aspect of the present 1$ {hQ afea CQ in fadius at differeQt ^ 

For ease of explanation but without any limitation intended * alternative i V) the circle may exhibit a constant 

thereby the example of FIG 4 is described m the context of ^ k h to ^ actual iflt ^ ^ 

the hardware environment described above in FIGS. 1-2. rfbed probabilit y at any point on the display 120. Of 

The process 400 is initiated in step 402. As an example, this c ordinarily ski i led artisans having the benefit of this 

may occur automatically when me computer 102 boots-up, 2Q disclosure wiU recognize a number of other shapes and 

under control of one ;of the application programs 112 when configurations of gaze arca without departing from this 

the operator manually activates the integration module 114, invention 

or at another time. In the illustrated example, when the , , . , , ^ , . 

routine 400 begins, display of the cursor associated with the Abo in step 408, the integration module 114 determines 

user input device 106 is suppressed ("hidden"). „ 'j 16 im ?^ oa of in 5. al mou f c m °J emen <> " 

T f. ( , r p . . „ 25 detected m step 406. This may be performed, for example, 

Tracking Gaze Position by process i Qg tne m0 use 's output signals to develop a mouse 

In response to step 402, the integration module 114 starts movement vector. This vector represents the average direc- 

to monitor the operator's gaze position in step 404, The gaze Uon 0 f mouse movement, and can be calculated in a matter 

position is a point where the gaze tracking apparatus 104 and 0 f milliseconds. 

gaze tracking module 118 calculate the operator's actual 30 Hayi identified the direc ti 0D of initial mouse 

gaze point to be. This calculated point may include some movementj lhe integration modu i e 114 identifies the 

error due to the limits of resolution of the gaze tracking approximatc ^ntcr of the gaze area, and computes a line 

apparatus 104, intrinsic difficulties m calculating gaze (e.g., jectm outward from the approximate center, opposite 

accounting for head movement in corneal reflection systems, ^ dircction of me mouse movement vector. With this 

etc.), and other sources of error. TTiese sources of error are 35 Mormatioi ^ the integration module 114 identifies a point 

collectively referred to as system noise , and may be whcre ^ linc frQm ^ arca , & cenlef intcrsects thc 

understood by studying and measuring the operation of the ih of me are a. This is the initial display position, 

system 100. For example, it may be determined in some ^ ^ ^ &[ k ^ fc itkmed ^ thal , if the 

systems that the error between gaze position and actual gaze operator continues to move the mouse ia the same fa^on 

point has a Gaussian distribution. 40 a& detected> me mouse will travel ^ tne gaze area 

As an example, step 404 may be performed by receiving m $ toward its center. Thus, the initial display position is 

x-y position signals from the gaze tracking module 118. placed on the opposite side of the gaze area from the 

Detecting User Input direction the mouse is initially moved. As an example, if the 

In step 406, the integration module 114 determines user input of step 406 comprises movement toward the top 

whether there has been any manual user input from the 45 of the display 120, the initial display position is set at the 

device 106. In other words, step 406 determines whether the bottom periphery of the gaze area. 

input device 106 has been mechanically activated by the Having computed the initial display position in step 408, 

user. In the present example, step 406 senses whether the tnc integration module 114 displays the cursor there. The 

operator has moved the mouse 110 across its resting surface, ^ cursor having been suppressed, this initial appearance of the 

such as a mouse pad. In a system where a trackball is used cursor is called "warping". Accordingly, in step 408 the 

instead of the mouse 110, step 406 senses whether the ball cursor is said to have "warped" to its initial display position, 

has been rolled. ^ Qne 0 p(j onj one or more highly visible animations may 

Responding to User Input be used to help the operator find the warped cursor in its 

If movement is detected, the integration module 114 55 initial display position. Namely, the cursor may exhibit a 

computes an "initial display position" and displays the brief animation (known as "spriting") to catch the operator's 

cursor at this position in step 408. In an alternative eye and enable the operator to quickly reacquire the cursor, 

embodiment, initial display position may be computed One example of this animation comprises a briefly appearing 

continually, periodically, or according to another schedule sequence of increasingly smaller circles, drawn concentric 

independent of the presence of operator input. 60 to the cursor's initial display position. Since the human 

In the illustrated example, the initial display position is visual system is very sensitive to motion, the animated 

computed by the following sub-steps. Although these steps cursor is quickly found by the operator. After the animation 

are described in an order that represents one embodiment of completes, the operator has a visual lock on the cursor and 

the invention, this order may be changed in different ways proceeds to reposition the cursor normally with the input 

without departing from the scope of this invention. 65 device 106. 

First, a "gaze area" is calculated, comprising a region that As an alternative to the initial display calculation 

surrounds the gaze position at the time manual user input is described above (step 408), the initial display position may 
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have a fixed relationship with respect to the gaze area. For sensed, such as clicking one or more mouse buttons, striking 

instance, the initial display position may always be at the a keyboard key, etc. Despite the end and renewal of user 

bottom of the gaze area, or another area such as the top, activity, the gaze tracking apparatus 104 and gaze tracking 

top-right corner, center, etc. Some users may benefit from a module 118 continue to cooperatively follow the operator's 

static initial display position, due to its consistency. Since 5 5 aze > m ^ periodically recalculate the current gaze position, 

the user can always anticipate the cursor's initial position, In response to the renewed activity, the routine 400 

the user can consistently initiate cursor movement in the progresses from step 416 to step 418, in which the mtegra- 

same direction t * 0D mo ° u l e H** determines whether the current gaze posi- 

... 1( , tion is in the same vicinity as the position of step 414. Also, 

As another initial display position alternative, the selec- ^ module m k ^ ^ sam / positioQj as 

tion of the initial display position may also consider video 10 shoWD by step 420 . Alternatively, if the cursor was hidden in 

screen content as input. For example, if the calculated gaze step 414> then me mo dule 114 displays the cursor in the 

area only includes one target (such as an icon), the initial position from which it was hidden. Following step 420, 

display position may be placed over that target. If the gaze control passes to step 410 and continues with the routine 400 

area includes multiple targets, the initial display position a s discussed above. 

may consider a predetermined default target, a historically 15 However, if step 418 finds the user's current gaze area at 

most-frequent target, a position between competing targets the time of renewed activity differs from the cursor's resting 

of equal status, the target most aligned with the initial position from step 414, then control advances to step 408, 

direction of the user input device, etc. which computes a new initial display position for the cursor, 

After step 408, the integration module 114 directs normal and war P s the cursor accordingly, as described above. Fol- 

movement of the cursor according to user input through the 20 lowing step 408, the routine 400 continues as discussed 

input device 106, irrespective of eye gaze. above. 

In one embodiment of the invention, the system 100 may OTHER EMBODIMENTS 

be implemented to automatically recalibrate the gaze track- Wm j e mc foregoing disclosure shows a number of illus- 

ing module 118. Namely, if the operator selects a target in trative embodiments of the invention, it will be apparent to 

the gaze area, the selected target is assumed to be the actual those skilled in the art that various changes and modifica- 

gaze point. The predicted gaze position and the position of tions can be made herein without departing from the scope 

the selected target are sent to the gaze tracking module 118 of the invention as defined by the appended claims, 

as representative "new data" for use in recalibration. The Furthermore, although elements of the invention may be 

gaze tracking module 118 may use the new data to recali- described or claimed in the singular, the plural is contem- 

brate the gaze direction calculation. The integration module 30 plated unless limitation to the singular is explicitly stated. 

114 may also use this data to update the calculation of the What is claimed is: 

gaze area on the display 120. The recalibration may com- 1. A computer-implemented method for eye gaze assisted 
pensate for many different error sources. For example, cursor positioning a video screen, comprising: 
recalibration may be done per user or video display, or for ^ operating a gaze tracking apparatus to monitor eye ori- 
different operating conditions such as indoor use, outdoor entation of a computer operator as the operator views a 
use, stationary/moving system operation, etc. Regardless of video screen and compute a measured gaze position of 
the way the new data is used by the gaze tracking apparatus the operator upon the video screen according to the 
104, the new data may also be used by the integration monitored eye orientation, and concurrently monitor- 
module 114 to estimate the size and shape of the gaze area ing a operator input device for mechanical activation by 
on the display 120. For example, in the system 100, the the operator; 

standard deviation of error can be estimated and updated according to the operator's eye orientation, determining a 
according to the new data. The gaze area may also be gaze area comprising a sub -region of the video screen 
estimated independently by the application programs 112. including the measured gaze position; and 
For purposes of recalibration and gaze area estimation, the ^ suspending display of a cursor until detection of activa- 
system 100 and the gaze tracking apparatus 104 may main- tion 0 f me operator input device, and thereupon co at- 
tain and save history and statistics of the new data. This puting an initial cursor display position within the gaze 
allows profiles to be created and restored for each user, area an( j displaying the cursor upon the video screen at 
system, operating condition, etc. the initial cursor display position, and then moving the 
User Inactivity SQ cursor across the video screen in accordance with the 
Normal cursor movement continues in step 410 until the activation of the operator input device irrespective of 
integration module 114 detects user inactivity in step 412. gaze. 

User inactivity may be defined by various conditions, such 2. The method of claim 1, the determining of the gaze area 

as absence of mouse input for a predetermined time, such as comprising: 

100 milliseconds. As another option, inactivity may consti- ss calculating a sub -region of the video screen having a 

tute the absence of any input from all components of the user predetermined probability of including an actual gaze 

input device 106. In response to user in activity, the inte- point of the operator. 

gration module 114 keeps displaying the cursor in the same 3. The method of claim 2, the subregion having a sub- 

place, as shown by step 414. As an alternative, the module stantially circular shape. 

114 may respond to the detected inactivity by hiding the 60 4. The method of claim 1, the subregion having a sub- 

cursor. stantially fixed shape and size, 

Sensing and Processing Renewed Activity 5. The method of claim 1, the computing of the initial 

The module 114 then monitors the user input device 106 display position comprising: 

for renewed activity in step 416. In the illustrated identifying a target in the gaze area; 

embodiment, renewed activity comprises movement of the 65 determining a movement direction represented by the 

mouse 110, representing a horizontal and/or vertical cursor detected mechanical activation of the operator input 

movement. However, other types of renewed activity may be device; and 
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establishing the display position at the periphery of the 
gaze area at a location where continued movement of 
the cursor in the movement direction would intersect 
the identified target 

6. The method of claim 1, the computing of the initial 5 
display position comprising: 

establishing the initial display position at a predetermined 
position on a periphery of the gaze area. 

7. The method of claim 6, the predetermined position 
being the bottom of the gaze area, as viewed by the operator. 10 

8. The method of claim 1, the operator input device 
comprising a computer mouse. 

9. The method of claim 8, the mechanical activation of the 
operator input device comprising movement of the mouse. 

10. The method of claim 1, the determining of the gaze 15 
area being performed in response to the detection of 
mechanical activation of the operator input device. 

11 . The method of claim 1, the measured gaze position 
being computed by a gaze tracking module, the method 
further comprising, after commencing the step of moving 20 
the cursor on the video screen in accordance with the 
activation of the operator input device, providing a calibra- 
tion output, comprising: 

receiving selection of a target by activation of the operator 
input device, the target having a target position; and 25 

directing the measured gaze position and the target posi- 
tion to the gaze tracking module, and then the gaze 
tracking module performing recalibration responsive to 
the measured gaze position and target position .to 3Q 
improve accuracy of future computations of measured 
gaze position. 

12. The method of claim 1, the computation of the initial 
display position comprising: 

determining how many targets reside in the gaze area; and 35 
if a single target resides in the gaze area, establishing the 
initial display position within the single target. 

13. The method of claim 1, the operating of the gaze 
tracking apparatus and determining of a current gaze posi- 
tion being performed substantially continuously. 40 

14. The method of claim 1, the computation of the initial 
display position comprising: 

recognizing a target in the gaze area predesignated as a 

default target; and 
establishing the initial display position within the default 45 

target. 

15. The method of claim 1, the computation of the initial 
display position comprising: 

evaluating targets in the gaze area with respect to fre- 
quency of past selection; and 50 

establishing the initial display position within a target 
with most frequent past selection. 

16. The method of claim 1, the computation of the initial 
display position comprising: 55 

determining how many targets reside in the gaze area; and 
if multiple targets reside in the gaze area, establishing the 
initial display position between the targets. 

17. A computer-implemented method for eye gaze 
assisted cursor positioning a video screen, comprising: 

operating a gaze tracking apparatus to monitor eye ori- 
entation of a computer operator as the operator views a 
video screen and compute a measured gaze position of 
the operator upon the video screen according to the 
monitored eye orientation, and concurrently monitor- 65 
ing a operator input device for mechanical activation by 
the operator; 
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according to the operator's eye orientation, determining a 
gaze area comprising a sub -region of the video screen 
including the measured gaze position, the determining 
of the gaze area being performed repeatedly according 
to a predetermined schedule irrespective of detection of 
mechanical activation of the operator input device; 

upon detection of activation of the operator input device, 
computing an initial display position within the gaze 
area and displaying a cursor upon the video screen at 
the initial display position, and then moving the cursor 
on the video screen in accordance with the activation of 
the operator input device irrespective of gaze. 

18. A computer-implemented method for eye gaze 
assisted cursor positioning' a video screen, comprising: 

operating a gaze tracking apparatus to monitor eye ori- 
entation of a computer operator as the operator views a 
video screen and compute a measured gaze position of 
the operator upon the. video screen according to the 
monitored eye orientation, and concurrently monitor- 
ing a operator input device for mechanical activation by 
the operator; 

according to the operator's eye orientation, determining a 
gaze area comprising a sub-region of the video screen 
including the measured gaze position; and the deter- 
mining of the gaze area being performed repeatedly 
according to a predetermined schedule irrespective of 
detection of mechanical activation of the operator input 
device; 

upon detection of activation of the operator input device, 
computing an initial display position within the gaze 
area and displaying a cursor upon the video screen at 
the initial display position, and then moving the cursor 
on the video screen in accordance with the activation of 
the operator input device irrespective of gaze; 
detecting mechanical inactivity of the operator input 
device for a predetermined time, the cursor having an 
ending position on the video screen; 
in response to detecting the mechanical activity, 

monitoring of the operator input device for mechanical 

activation by the operator; 
operating the gaze tracking apparatus to monitor eye 
orientation of the computer operator as the operator 
views the video screen and compute measured gaze 
position of the operator upon the video screen 
according to the monitored eye orientation; 
determining a current gaze area comprising a sub- 
region of the video screen including the measured 
gaze position; 

upon detection of activation of the operator input device, 
determining a difference between the cursor's ending 
position and a most recently calculated gaze position; 
and 

applying predetermined criteria to the difference, and 
if the difference satisfies the criteria, repositioning the 
cursor at the ending position and then moving the 
cursor on the video screen in accordance with the 
activation of the operator input device irrespective of 
gaze; 

if the difference fails the criteria, computing an initial 
display position within the gaze area and displaying 
the cursor upon the video screen at the initial display 
position, and then moving the cursor on the video 
screen in accordance with the activation of the 
operator input device irrespective of gaze. 

19. The method of claim 18, following displaying of the 
cursor upon the video screen at the initial display position, 
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ceasing the operation of the gaze tracking apparatus and determining how many targets reside in the gaze area; and 

determining of the gaze area. if multiple targets reside in the gaze area, establishing the 

20. A signal-bearing medium tangibly embodying a pro- initial display position between the targets. 

gram of machine-readable instructions executable by a digj- 32. The medium of claim 20, the operating of the gaze 

tal processing apparatus to perform a method for computer- 5 tracking apparatus and determining of a current gaze posi- 

implemented eye gaze assisted cursor positioning on a video lion being performed substantially continuously, 

screen, said method comprising: 33. The medium of claim 20, the measured gaze position 

operating a gaze tracking apparatus to monitor eye ori- J? 6 "* computed by a gaze tracking module, the method 

entation of a computer operator as the operator views a ^rther composing, after commencing the step of moving 

video screen and compute a measured gaze position of 10 the cursor on the video screen m accordance with the 

. , • • i j. , activation of the operator input device, providing a calibra- 
the operator upon the video screen according. to the 

.. , . . .. , t1 Uon output, comprising: 

monitored eye orientation, and concurrently monitor- . \ . \. % , 4 , . t . etU 

/ . tJ . - i . i .* i receiving selection of a target by activation of the operator 

ing a operator input device for mechanical activation by . . °. . iU . A • * * •*• j 

the o erator mpU ^ evice ' ^ c tar 5 et having a target position; and 

^ / , . . , , . _ directing the measured gaze position and the target posi- 

according to the operator s eye orientation determining a 15 ^ to ^ ^ ^ ^ ^ ^ 

gaze area composing a sub-region of the video screen tracking module performing recalibration responsive to 

including the measured gaze position; and me mcasurc d gaze position and target position to 

suspending display of a cursor until detection of activa- improve accuracy of future computations of measured 

tion of the operator input device, and thereupon com- gaze position. 

puting an initial cursor display position within the gaze 20 34. The medium of claim 20, the computation of the initial 

area and displaying the cursor upon the video screen at display position comprising: 

the initial cursor display position, and then moving the determining how many targets reside in the gaze area; and 

cursor across the video screen in accordance with the if a single target resides in the gaze area, establishing the 

activation of the operator input device irrespective of initial display position within the single target, 

gaze. 25 35. The medium of claim 20, the computation of the initial 

21. The medium of claim 20, the determining of the gaze display position comprising: 

area comprising: recognizing a target in the gaze area predesignated as a 

calculating a sub-region of the video screen having a default target; and 

predetermined probability of including an actual gaze establishing the initial display position within the default 

point of the operator. target. 

22. The medium of claim 21, the sub-region having a 3<j. a signal-bearing medium tangibly embodying a pro- 
substantially circular shape. gram 0 f machine-readable instructions executable by a digj- 

23. The medium of claim 20, the sub-region having a ta i processing apparatus to perform a method for computer- 
substantially fixed shape and size. implemented eye gaze assisted cursor positioning on a video 

24. The medium of claim 20, the computing of the initial screen, said method comprising: 

display position comprising: operating a gaze tracking apparatus to monitor eye ori- 
idenufying a target in the gaze area; entation of a computer operator as the operator views a 
determining a movement direction represented by the video screen and compute a measured gaze position of 
detected mechanical activation of the operator input 4Q the operator upon the video screen according to the 
device; and monitored eye orientation, and concurrently monitor- 
establishing the display position at the periphery of the ing a operator input device for mechanical activation by 
gaze area at a location where continued movement of the operator;. 

the cursor in the movement direction would intersect according to the operator's eye orientation, determining a 

the identified target. 45 gaze area comprising a sub -region of the video screen 

25. The medium of claim 20, the computing of the initial including the measured gaze position; the determining 
display position comprising: of the gaze area being performed repeatedly according 

establishing the initial display position at a predetermined to a predetermined schedule irrespective of detection of 

position on a periphery of the gaze area. mechanical activation of the operator input device; 

26. The medium of claim 25, the predetermined position 50 upon detection of activation of the operator input device, 
being the bottom of the gaze area, as viewed by the operator. computing an initial display position within the gaze 

27. The medium of claim 20, the operator input device area and displaying a cursor upon the video screen at 
comprising a computer mouse. the initial display position, and then moving the cursor 

28. The medium of claim 27, the mechanical activation of on the video screen in accordance with the activation of 
the operator input device comprising movement of the 55 the operator input device irrespective of gaze, 
mouse. 37. A signal -bearing medium tangibly embodying a pro- 

29. The medium of claim 20, the determining of the gaze gram of machine-readable instructions executable by a digi- 
area being performed in response to the detection of tal processing apparatus to perform a method for computer- 
mechanical activation of the operator input. implemented eye gaze assisted cursor positioning on a video 

30. The medium of claim 20, the computation of the initial 60 screen, said method comprising: 

display position comprising: operating a gaze tracking apparatus to monitor eye ori- 

evaluating targets in the gaze area with respect to fre- entation of a computer operator as the operator views a 

quency of past selection; and video screen and compute a measured gaze position of 

establishing the initial display position within a target the operator upon the video screen according to the 

with most frequent past selection. 65 monitored eye orientation, and concurrently monitor- 

31. The medium of claim 20, the computation of the initial ing a operator input device for mechanical activation by 
display position comprising: the operator; 
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according to the operator's eye orientation, determining a 
gaze area comprising a sub-region of the video screen 
including the measured gaze position; 
upon detection of activation of the operator input device, 
computing an initial display position within the gaze S 
area and displaying a cursor upon the video screen at 
the initial display position, and then moving the cursor 
on the video screen in accordance with the activation of 
the operator input device irrespective of gaze; 
detecting mechanical inactivity of the operator input 10 
device for a predetermined time, the cursor having an 
ending position on the video screen; 
in response to detecting the mechanical inactivity, 
monitoring of the operator input device for mechanical 

activation by the operator; 15 
operating the gaze tracking apparatus to monitor eye 
orientation of the computer operator as the operator 
views the video screen and compute measured gaze 
position of the operator upon the video screen 
according to the monitored eye orientation; 20 
determining a current gaze area comprising a sub- 
region of the video screen including the measured 
gaze position; 

upon detection of activation of the operator input device, 
determining a difference between the cursor's ending 25 
position and a most recently calculated gaze position; 
and 

applying predetermined criteria to the difference, and 
if the difference satisfies the criteria, repositioning the 3Q 
cursor at the ending position and then moving the 
cursor on the video screen in accordance with the 
activation of the operator input device irrespective of 
gaze; 

if the difference fails the criteria, computing an initial 35 
display position within the gaze area and displaying 
the cursor upon the video screen at the initial display 
position, and then moving the cursor on the video 
screen in accordance with the activation of the 
operator input device irrespective of gaze. 4Q 

38. The medium of claim 37, following displaying of the 
cursor upon the video screen at the initial display position, 
ceasing the operation of the gaze tracking apparatus and 
determining of the gaze area. 

39. A computer-driven eye gaze assisted cursor position- ^ 
ing system, comprising: 

a gaze tracking apparatus to monitor eye orientation of a 
computer operator as the operator views a video screen 
and compute a measured gaze position of the operator 
upon the video screen according to the monitored eye 5Q 
orientation; 

a gaze tracking module to use the operators eye orienta- 
tion to determine a gaze area comprising a sub-region 
of the video screen including the measured gaze posi- 
tion; 55 

an operator input device, responsive to mechanical acti- 
vation by a computer operator; 

an operator input module monitoring the operator input 
device for mechanical activation; and 

an integration module programmed to suspend display of 60 
a cursor until detection of activation of the operator 
input device, and thereupon to compute an initial cursor 
display position within the gaze area and display the 
cursor upon the video screen at the initial cursor display 
position, and then move the cursor across the video 65 
screen in accordance with the activation of the operator 
input device irrespective of gaze. 
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40. The system of claim 39, the determining of the gaze 
area comprising: 

calculating a sub-region of the video screen that includes 
an actual gaze point of the operator with a predeter- 
mined probability. 

41. The system of claim 40, the sub-region having a 
substantially circular shape. 

42. The system of claim 39, the sub-region having a 
substantially fixed shape and size. 

43. The system of claim 39, the computing of the initial 
display position comprising: 

identifying a target in the gaze area; 

determining a movement direction represented by the 

detected mechanical activation of the operator input 

device; and 

establishing the display position at the periphery of the 
gaze area at a location where continued movement of 
the cursor in the movement direction would intersect 
the identified target. 

44. The system of claim 39, the computing of the initial 
display position comprising: 

establishing the initial display position at a predetermined 
position on a periphery of the gaze area. 

45. The system of claim 44, the predetermined position 
being the bottom of the gaze area, as viewed by the operator. 

46. The system of claim 39, the operator input device 
comprising a computer mouse. 

47. The system of claim 46, the mechanical activation of 
the operator input device comprising movement of the 
mouse. 

48. The system of claim 39, the determining of the gaze 
area being performed in response to the detection of 
mechanical activation of the operator input device. 

49. The system of claim 39, the computation of the initial 
display position comprising: 

recognizing a target in the gaze area predesignated as a 

default target; and 
establishing the initial display position within the default 

target. 

50. The system of claim 39, the computation of the initial 
display position comprising: 

evaluating targets in the gaze area with respect to fre- 
quency of past selection; and 

establishing the initial display position within a target 
with most frequent past selection. 

51. The system of claim 39, the operating of the gaze 
tracking apparatus and determining of a current gaze posi- 
tion being performed substantially continuously. 

52. The system of claim 39, the computation of the initial 
display position comprising: 

determining how many targets reside in the gaze area; and 
if multiple targets reside in the gaze area, establishing the 
initial display position between the targets, 

53. The system of claim 39, the integration module being 
further programmed to provide a calibration output, the 
providing of the calibration output comprising: 

after commencing the step of moving the cursor on the 
video screen in accordance with the activation of the 
operator input device; 

receiving selection of a target by activation of the 
operator input device, the target having a target 
position; and 

directing the measured gaze position and the target 
position to the gaze tracking module, and then the 
gaze tracking module performing recalibration 
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responsive to the measured gaze position and target 
position to improve accuracy of future computations 
of measured gaze position. 

54. The system of claim 39, the computation of the initial 
display position comprising: 5 

determining how many targets reside in the gaze area; and 
if a single target resides in the gaze area, establishing the 
initial display position within the single target. 

55. A computer-driven eye gaze assisted cursor position- 
ing system, comprising: 10 

a gaze tracking apparatus to monitor eye orientation of a 
computer operator as the operator views a video screen 
and compute a measured gaze position of the operator 
upon the video screen according to the monitored eye J5 
orientation; 

a gaze tracking module to use the operator's eye orien- 
tation to determine a gaze area comprising a sub-region 
of the video screen including the measured gaze 
position, the gaze tracking module determining the 2 o 
gaze area repeatedly according to a predetermined 
schedule irrespective of detection of mechanical acti- 
vation of the operator input device; 

an operator input device, responsive to mechanical acti- 
vation by a computer operator; 25 

an operator input module monitoring the operator input 
device for mechanical activation; and 

an integration module programmed to compute, upon 
detection of activation of the operator input device, 
compute an initial display position within the gaze area 30 
and display a predefined cursor upon the video screen 
at the initial display position, and then move the cursor 
on the video screen in accordance with the activation of 
the operator input device irrespective of gaze. 

56. A computer-driven eye gaze assisted cursor position- 35 
ing system, comprising: 

a gaze tracking apparatus to monitor eye orientation of a 
computer operator as the operator views a video screen 
and compute a measured gaze position of the operator 4Q 
upon the video screen according to the monitored eye 
orientation; 

a gaze tracking module to use the operator's eye orien- 
tation to determine a gaze area comprising a sub-region 
of the video screen including the measured gaze posi- 45 
tion; 

an operator input device, responsive to mechanical acti- 
vation by a computer operator; 

an operator input module monitoring the operator input 
device for mechanical activation; and 50 

an integration module programmed to compute, upon 
detection of activation of the operator input device, an 
initial display position within the gaze area and display 
a predefined cursor upon the video screen at the initial 
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display position, and then move the cursor on the video 

screen in accordance with the activation of the operator 

input device irrespective of gaze; 
the integration module additionally being programmed to 

perform operations comprising: 

detecting mechanical inactivity of the operator input 
device for a predetermined time, the cursor having 
an ending position on the video screen; 

upon detection of re-activation of the operator input 
device by the operator, determining a difference 
between the ending position and a current gaze 
position computed by the gaze tracking apparatus; 
and 

applying predetermined criteria to the difference, and 
if the difference satisfies the criteria, repositioning 
the cursor at the ending position and then moving 
the cursor on the video screen in accordance with 
the activation of the operator input device irre- 
spective of gaze; 
if the difference fails the criteria, computing an initial 
display position within the gaze area and display- 
ing the cursor upon the video screen at the initial 
display position, and then moving the cursor on 
the video screen in accordance with the activation 
of the operator input device irrespective of gaze. 

57. The system of claim 56, following displaying of the 
cursor upon the video screen at the initial display position, 
ceasing the operation of the gaze tracking apparatus and 
determining of the gaze area. 

58. A computer-driven eye gaze assisted cursor position- 
ing system, comprising: 

first gaze tracking means monitoring eye orientation of a 
computer operator as the operator views a video screen 
and computing a measured gaze position of the opera- 
tor upon the video screen according to the monitored 
eye orientation; 

second gaze tracking means for using the operator's eye 
orientation to determine a gaze area comprising a 
sub-region of the video screen including the measured 
gaze position; 

first operator input means for receiving mechanical acti- 
vation by a computer operator; 

second operator input means for monitoring the first 
operator input means for mechanical activation; and 

integration means for suspending display of a cursor until 
detection of activation of the operator input device, and 
thereupon computing an initial cursor display position 
within the gaze area and displaying the cursor upon the 
video screen at the initial cursor display position, and 
then moving the cursor across the video screen in 
accordance with the activation of the operator input 
device irrespective of gaze. 

***** 
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